Digital ink processing system, method, and program

ABSTRACT

A digital ink processing system, method, and program are provided that are capable of presenting, to a user, useful and highly relevant information as a search result when a search is performed using digital ink. A processor included in a tablet enables a pointing operation of an electronic pen relative to a stroke or strokes. The processor, after enabling the pointing operation of the electronic pen, performs a search for content related to a semantic attribute of the stroke or strokes pointed at, or requests an external server to perform the search. The processor performs control so as to display the related content retrieved by the search on a display with the stroke or strokes.

BACKGROUND Technical Field

The present disclosure relates to a digital ink processing system,method, and program for processing digital ink.

Description of the Related Art

Patent Document 1 discloses a technique of performing a search bysetting as a search key at least a portion of information displayed on ascreen of an information processing apparatus such as a personalcomputer, and display a search result in a pop-up window on the screen.

PRIOR ART DOCUMENT Patent Document

-   Patent Document 1: Japanese Patent Laid-Open No. 2015-114955

SUMMARY OF DISCLOSURE Technical Problems

In digital ink representative of a set of strokes, strokes often conveymeaning only as a group. However, when a user, while editing the digitalink, points to a section to set it as a search key to attempt a search,the search key may fail to convey any meaning or the search key mayconvey wrong meaning, resulting in a failure to return appropriatesearch results.

An aspect of the present disclosure is to provide a digital inkprocessing system, method, and program that are capable of presenting,to a user, useful and highly relevant information as a search resultwhen a search is performed using digital ink.

Technical Solution

According to a first aspect of the present disclosure, a digital inkprocessing system includes an electronic pen, and a tablet thatgenerates digital ink describing a stroke or strokes according to aninput made by the electronic pen through a display. The tablet has aprocessor, and the processor enables a pointing operation of theelectronic pen relative to the stroke or strokes. After enabling thepointing operation of the electronic pen relative to the stroke orstrokes, the processor performs a search for content related to asemantic attribute of the stroke or strokes pointed at, or requests anexternal server to perform the search, and performs control so as todisplay content retrieved by the search on the display with the strokeor strokes.

Further, the processor may perform control so as to cause the stroke orstrokes for which the pointing operation has been enabled to behighlighted on the display as compared to before the enablement.

Further, the processor may enable the pointing operation of theelectronic pen relative to a stroke or strokes to which a semanticattribute has been assigned.

The system may further include a digital ink server that analyzes thedigital ink transmitted from the tablet and assigns the semanticattribute to the stroke or strokes. The processor may enable thepointing operation of the electronic pen relative to a stroke or strokesfor which semantic data representative of the semantic attribute hasbeen acquired from the digital ink server.

Further, the processor may enable the pointing operation of theelectronic pen relative to a stroke or strokes on which a user operationof adding a mark or an annotation has been performed.

The system may further include a content server that stores content inassociation with use IDs (use identifications). The processor maytransmit data, which includes the use ID and the semantic attribute, tothe content server and request a search to acquire, from the contentserver, content that relates to the semantic attribute and that isusable.

Further, the processor may acquire a different one of the contentdepending on the degree of occurrence of the semantic attribute based onthe same use ID.

Further, the use ID may be a pen ID that identifies the electronic pen.

Further, in a case where the digital ink is generated using a firstelectronic pen that is assigned a first pen ID, while a secondelectronic pen is used that is assigned a second pen ID different fromthe first pen ID, the processor may prohibit or restrict editing of thedigital ink while accepting the pointing operation.

Further, the content server may perform a charging process according tousage of content, for a user of the content and a business entity thatprovides the content.

According to a second aspect of the present disclosure, a digital inkprocessing method executed by a processor is provided, wherein theprocessor is configured to generate digital ink describing a stroke orstrokes according to an input made by an electronic pen through adisplay. The method includes enabling a pointing operation of theelectronic pen relative to the stroke or strokes. The method includes,after enabling the pointing operation of the electronic pen relative tothe stroke or strokes, performing a search for content related to asemantic attribute of the stroke or strokes pointed at, or requesting anexternal server to perform the search. The method includes performingcontrol so as to display content retrieved by the search on the displaywith the stroke or strokes.

According to a third aspect of the present disclosure, a digital inkprocessing program is provided that causes a processor configured togenerate digital ink describing a stroke or strokes according to aninput made by an electronic pen through a display to execute: a step ofenabling a pointing operation of the electronic pen relative to thestroke or strokes, a step of, after enabling the pointing operation ofthe electronic pen relative to the stroke or strokes, performing asearch for content related to a semantic attribute of the stroke orstrokes pointed at, or requesting an external server to perform thesearch, and a step of performing control so as to display contentretrieved by the search on the display with the stroke or strokes.

BRIEF SUMMARY

The present disclosure makes it possible to present, to a user, usefuland highly relevant information as a search result when a search isperformed using digital ink.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is an overall configuration diagram of a digital ink processingsystem according to an embodiment of the present disclosure.

FIG. 2 is a block diagram of a digital ink server, a content server, anda user terminal illustrated in FIG. 1.

FIG. 3 is a sequence diagram concerning a semantic interpretationoperation of the digital ink processing system.

FIG. 4A is a schematic diagram illustrating a set of strokes in alegible form.

FIG. 4B is a diagram illustrating an example of a data structure ofdigital ink.

FIG. 5A is a diagram illustrating a result of grouping of the strokes.

FIG. 5B is a diagram illustrating a result of assignment of semanticattributes to groups obtained by the grouping in FIG. 5A.

FIG. 6A is a transition diagram illustrating a first state change in adisplay of the user terminal.

FIG. 6B is a transition diagram illustrating the first state change inthe display of the user terminal.

FIG. 7A is a transition diagram illustrating a second state change inthe display of the user terminal.

FIG. 7B is a transition diagram illustrating the second state change inthe display of the user terminal.

FIG. 8 is a sequence diagram concerning content citation operation ofthe digital ink processing system.

FIG. 9A is a diagram illustrating an example of the data structure ofelectronic pen information.

FIG. 9B is a diagram illustrating an example of the data structure ofcontent management information.

FIG. 10A is a transition diagram illustrating a third state change inthe display of the user terminal.

FIG. 10B is a transition diagram illustrating the third state change inthe display of the user terminal.

FIG. 11A is a diagram illustrating an example operation of a userterminal different from a user terminal on which handwriting input isperformed.

FIG. 11B is a diagram illustrating an example operation of a userterminal different from the user terminal on which handwriting input isperformed.

FIG. 12A is a diagram illustrating a first refined example of theoperation of the digital ink processing system.

FIG. 12B is a diagram illustrating the first refined example of theoperation of the digital ink processing system.

FIG. 13A is a diagram illustrating a second refined example of theoperation of the digital ink processing system.

FIG. 13B is a diagram illustrating the second refined example of theoperation of the digital ink processing system.

FIG. 14 is a schematic diagram illustrating a first example of a flow ofcharging made at the time of a learning support service.

FIG. 15 is a schematic diagram illustrating a second example of the flowof charging made at the time of the learning support service.

DETAILED DESCRIPTION

[Configuration of Digital Ink Processing System 10]

FIG. 1 is an overall configuration diagram of a digital ink processingsystem 10 according to an embodiment of the present disclosure. FIG. 2is a block diagram of a digital ink server 20, a content server 30, anda user terminal 40 illustrated in FIG. 1. The digital ink processingsystem 10 is configured to be capable of providing a “learning supportservice” to allow a user, such as a student, to learn efficiently usingan electronic notebook. Specifically, the digital ink processing system10 includes the digital ink server 20, the content server 30, one ormore user terminals 40, and one or more electronic pens 50.

WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF(Ink Serialized Format) are exemplary data formats of digital ink (orink data), which are generally called “ink markup languages.” Varioustypes of software and programming languages can exchange data with eachother by using a data structure format of JSON (JavaScript® ObjectNotation) to describe such digital ink.

The digital ink server 20 is a computer that performs centralizedcontrol related to processing of digital ink Ink, and may be either of acloud type or an on-premises type. Here, the digital ink server 20 isdepicted as a single computer in the figure, but the digital ink server20 may alternatively be implemented as a group of computers that form adistributed system.

The digital ink server 20 specifically includes a communication unit 21,a control unit 22, and a storage unit 23. The communication unit 21 isan interface for transmitting and receiving electrical signals to andfrom an external device. The control unit 22 is formed of aprocessing/computing device including a CPU (Central Processing Unit) ora GPU (Graphics Processing Unit). The control unit 22 functions as a“processing engine” for the digital ink Ink by loading and executing aprogram stored in the storage unit 23. The storage unit 23 is formed ofa non-transitory computer-readable storage medium, such as an HDD (HardDisk Drive) or an SSD (Solid State Drive). Thus, the storage unit 23stores various data handled by the digital ink server 20.

The content server 30 is a computer that performs centralized controlrelated to provision of educational content including a textbook and adictionary, and may be either of the cloud type or the on-premises type.Here, the content server 30 is depicted as a single computer in thefigure, but the content server 30 may alternatively be implemented as agroup of computers that form a distributed system.

The content server 30 specifically includes a communication unit 31, acontrol unit 32, and a storage unit 33. The communication unit 31 is aninterface for transmitting and receiving electrical signals to and froman external device. The control unit 32 is formed of aprocessing/computing device including a CPU or a GPU. The control unit32 is configured to be capable of searching for and providing contentsuited for a user by loading and executing a program stored in thestorage unit 33.

The user terminal 40 is a terminal that functions as an electronicnotebook, and is formed of, for example, a tablet, a smart phone, or apersonal computer. The user terminal 40 specifically includes atouchscreen display 41, a touch IC (Integrated Circuit) 44, a displaydriver IC 45, a host processor 46, a memory 47, and a communicationmodule 48.

The touchscreen display 41 includes a display panel 42 capable ofoutputting content in a visible form, and sensor electrodes 43 arrangedon a display screen of the display panel 42 in an overlapping manner.The display panel 42 is capable of displaying a black-and-white image ora color image, and may be, for example, a liquid crystal panel or anorganic EL (Electro-Luminescence) panel. The sensor electrodes 43 areformed of a planar arrangement of an X-Y sensor coordinate systemincluding a plurality of X-line electrodes for sensing positions alongan X-axis and a plurality of Y-line electrodes for sensing positionsalong a Y-axis.

The touch IC 44 is an integrated circuit for performing drive control ofthe sensor electrodes 43. The touch IC 44 drives the sensor electrodes43 on the basis of control signals supplied from the host processor 46.The touch IC 44 thus implements a “pen detection function” of detectingthe state of the electronic pen 50, and a “touch detection function” ofdetecting a touch made by a finger or the like of a user.

The display driver IC 45 is an integrated circuit for performing drivecontrol of the display panel 42. The display driver IC 45 drives thedisplay panel 42 on the basis of image signals supplied from the hostprocessor 46 on a frame by frame basis. Images are thus displayed in adisplay area of the display panel 42. These images may include ahandwritten line drawn by the user using the electronic pen 50-as wellas an application window, an icon, and/or a cursor.

The host processor 46 is formed of a processing/computing deviceincluding an MPU (Micro-Processing Unit) or a CPU. The host processor 46performs a process of generating digital ink Ink using data from thetouch IC 44, an ink reproduction process for displaying a drawingrepresented by the digital ink Ink, and so on, by loading and executinga program stored in the memory 47.

The memory 47 is formed of a non-transitory computer-readable storagemedium. Here, the computer-readable storage medium is a portable medium,such as a magneto-optical disk, a ROM (Read Only Memory), a CD-ROM(Compact Disc-Read Only Memory), or a flash memory, or a storage device,such as an HDD or an SSD, contained in a computer system.

The communication module 48 is configured to be capable of transmittingand receiving electrical signals to and from an external device usingcable communication or wireless communication. Thus, the user terminal40 is capable of, for example, transmitting and receiving digital inkInk to and from the digital ink server 20, and receiving related contentC1 and C2 from the content server 30, through a network NW.

The electronic pen 50 is a pen-type pointing device and is configured tobe capable of performing one-way or two-way communication throughcapacitive coupling with the user terminal 40. The user is able to drawa picture or a character on the user terminal 40 by holding theelectronic pen 50 and moving the electronic pen 50 with a pen tip incontact with a touch surface of the touchscreen display 41. Theelectronic pen 50 is, for example, a stylus of an active capacitivecoupling type (AES (active electrostatic)) or an electromagneticinduction type (EMR (electro magnetic resource)).

[Operations of Digital Ink Processing System 10]

The digital ink processing system 10 according to this embodiment hasthe above-described configuration. Next, first and second operations ofthe digital ink processing system 10 will be described below, mainlyreferring to sequence diagrams of FIGS. 3 and 8. The “first operation”refers to an operation related to “semantic interpretation” ofinterpreting a collection of strokes and automatically assigning asemantic attribute (or a meaning attribute) to the collection ofstrokes. Meanwhile, the “second operation” refers to an operationrelated to “content citation” of timelily citing and displaying contentrelated to the semantic attribute.

<First Operation: Semantic Interpretation Operation>

First, the first operation of the digital ink processing system 10,i.e., a semantic interpretation operation, will now be described below.The first operation is implemented jointly by the user terminal 40 (morespecifically, the host processor 46) and the digital ink server 20 (morespecifically, the control unit 22).

In step S01 in FIG. 3, the user terminal 40 performs an authenticationusing identification information (hereinafter referred to as a pen ID)of the electronic pen 50. When a pen ID received and obtained from theelectronic pen 50 has been previously registered, the user terminal 40permits editing of digital ink Ink by the electronic pen 50. Meanwhile,when the pen ID has not been registered, the user terminal 40 does notpermit editing of the digital ink Ink by the electronic pen 50.

In step S02, the user terminal 40 accepts an input made using theelectronic pen 50 that has been permitted to make edits in step S01,wherein the input includes a variety of writing operations by the user.Examples of the writing operations include addition, deletion, andchange of strokes, selection of an icon, and addition of a mark or anannotation.

In step S03, the user terminal 40 updates the digital ink Ink, eitherperiodically or aperiodically, to reflect the writing operation acceptedin step S02 in the digital ink Ink. In the former case, the digital inkInk may be updated when a predetermined time has passed since the lastupdate time, for example. In the latter case, the digital ink Ink may beupdated when an update instruction operation by the user is received,for example.

In step S04, the user terminal 40 generates digital ink Ink in WILL(Ver. 3.0) format, for example, using data obtained through the writingoperation in step S02.

FIG. 4A is a schematic diagram illustrating a set G0 of strokes in avisible (legible) form. Here, the set G0 represents a handwrittenshopping list, and includes items of “milk,” “bread,” “eggs,” and“apples” arranged in order from top to bottom. FIG. 4B is a diagramillustrating an example of the data structure of the digital ink Ink.The digital ink Ink has a data structure in which [1] document metadata,[2] semantic data (ink semantics), [3] device data (devices), [4] strokedata (strokes), [5] grouping data (groups), and [6] context data(contexts) are arranged in order.

When the digital ink Ink is generated, the document metadata, the devicedata, the stroke data, and the context data have already beendetermined, but the semantic data and the grouping data are not yetdetermined. That is, (1) grouping of strokes and (2) assignment ofsemantic attributes, which will be described below, are not yetcompleted with respect to the digital ink Ink. Hereinafter, digital inkInk in which both the grouping data and the semantic data have NULLvalues may be referred to as “pre-assignment ink.”

In step S05, the user terminal 40 transmits the pre-assignment inkgenerated in step S04, with the pen ID associated therewith, to thedigital ink server 20.

In step S06, the digital ink server 20 receives the data transmittedfrom the user terminal 40 in step S05, thus acquiring the pre-assignmentink and the pen ID.

In step S07, the digital ink server 20 analyzes the stroke data includedin the pre-assignment ink acquired in step S06, and performs grouping ofthe strokes. Specifically, the digital ink server 20 groups the set G0of strokes into one or more groups (for example, five groups G1 to G5)on the basis of, for example, the order, positions, and shapes of thestrokes identified by the stroke data or the pen pressure of theelectronic pen 50. As a result of this grouping, grouping data isobtained that describes what stroke elements are included in what groupsor what groups are included in what groups.

In step S08, the digital ink server 20 performs a process of assigningsemantic attributes to the groups G1 to G5 of strokes obtained by thegrouping in step S07. Specifically, the digital ink server 20 uses adiscriminator (e.g., a hierarchical neural network) subjected to machinelearning to infer a semantic attribute of each of the groups G1 to G5and to assign the semantic attribute to each of the groups. As a resultof this process, grouping data is obtained that describes a semanticattribute of each group, wherein the semantic attribute includes a typeand a value in a pair.

For example, the above-mentioned discriminator is configured to receiveinput of feature amounts of the strokes (e.g., coordinates of startingpoints, intermediate points, and ending points, curvatures, etc.), andto output labels of semantic attributes. Examples of the “type” includea text (including a language type), a drawing (including a drawingtype), a mathematical equation, a chemical formula, a list, a table, andso on. Examples of the “value” include a handwritten character orcharacter string (e.g., “milk”), a name of a hand-drawn object (e.g.,“pencil” for an illustration of a pencil), and so on.

FIG. 5A is a diagram illustrating a result of the grouping of thestrokes. As will be understood from the figure, the set G0 includesgroup G1, and group G1 is made up of four groups G2 to G5. FIG. 5B is adiagram illustrating a result of the assignment of semantic attributesto the groups obtained by the grouping in FIG. 5A. The type and value ofgroup G1 are “LIST” and “NULL,” respectively. The type and value ofgroup G2 are “LIST ITEM” and “milk,” respectively. The type and value ofgroup G3 are “LIST ITEM” and “bread,” respectively. The type and valueof group G4 are “LIST ITEM” and “eggs,” respectively. The type and valueof group G5 are “LIST ITEM” and “apples,” respectively.

In step S09 in FIG. 3, the digital ink server 20 updates thepre-assignment ink such that the grouping data obtained in step S07 andthe semantic data obtained in step S08 are added to the pre-assignmentink. Hereinafter, the digital ink Ink with the grouping data and thesemantic data added thereto may be referred to as “post-assignment ink.”

In step S10, the digital ink server 20 transmits the post-assignment inkupdated in step S09 to the user terminal 40.

In step S11, the user terminal 40 receives the data transmitted from thedigital ink server 20 in step S10, thus acquiring the post-assignmentink including the grouping data and the semantic data.

In step S12, the user terminal 40 stores the post-assignment inkacquired in step S1 l in the memory 47.

In step S13, the user terminal 40 checks that a determination conditionconcerning the grouping of strokes is satisfied, and enables a pointingoperation relative to at least one group. An example of the“determination condition” is acquisition of semantic data from thedigital ink server 20. In this case, groups G2 to G5 for which thesemantic attributes have valid (other than NULL) values are chosen astargets of the enablement. It is to be noted that the enablement of thepointing operation is deferred for group G1, for which the semanticattribute has an invalid (NULL) value and thus no meaning is identified,or for one or more strokes that are in the midst of being written andthus have no meaning yet.

In step S14, the user terminal 40 causes groups G2 to G5 of strokes, forwhich the enablement has been performed in step S13, to be highlightedas compared to before the enablement. The “highlighting” means to changethe mode of display to a mode that makes visual recognition easier forthe user. Examples of highlighting include changing a display color,increasing a stroke width, adding an indication mark, and so on.

FIGS. 6A and 6B are transition diagrams illustrating a first statechange in a display of the user terminal 40. In more detail, FIG. 6Aillustrates the state of the display before the enablement, while FIG.6B illustrates the state of the display after the enablement. In theexample of FIG. 6B, underlines are added at four positions correspondingto groups G2 to G5 in the set G0 of strokes. This allows the user tounderstand at a glance that the pointing operation has been enabled forthe four words, “milk,” “bread,” “eggs,” and “apples.”

The first operation of the digital ink processing system 10 is thusfinished. Once the first operation is finished, the digital inkprocessing system 10 can start the second operation.

Although, in the example described above, the pointing operation isenabled when the user terminal 40 has acquired the semantic data fromthe digital ink server 20, the determination condition is not limited tothis example. For example, the determination condition may beacquisition of grouping data or semantic data through analysis of thedigital ink Ink by the user terminal 40 itself, or arrival of a timingat which the digital ink Ink is automatically or manually stored. Inaddition, the determination condition may be receipt by the userterminal 40 of an operation of adding a mark or an annotation to atleast a part of the set G0 of strokes.

FIGS. 7A and 7B are transition diagrams illustrating a second statechange in the display of the user terminal 40. As illustrated in FIG.7A, the user performs a writing operation of enclosing “eggs” with acircle (a so-called lasso) using the electronic pen 50 in a state inwhich no enablement has been performed for any part of the set G0 ofstrokes. As a result, along with the set G0 of strokes, a window 60 forchecking whether or not an enablement is necessary for “eggs” is newlydisplayed. Here, when an [ON] button in the window 60 is touched, thestate of the display of the user terminal 40 transitions from the stateof FIG. 7A to the state of FIG. 7B.

As illustrated in FIG. 7B, the word (eggs) corresponding to group G4 inthe set G0 of strokes is highlighted. This allows the user to understandat a glance that the pointing operation has been enabled only for theword “eggs.”

<Second Operation: Content Citation Operation>

Next, the second operation of the digital ink processing system 10, thatis, the content citation operation, will now be described below. Thesecond operation is implemented jointly by the user terminal 40 (morespecifically, the host processor 46) and the content server 30 (morespecifically, the control unit 32).

In step S21 in FIG. 8, the user terminal 40 accepts an operation (i.e.,the pointing operation) of pointing at one of groups G2 to G5, for whichthe enablement has been performed, in the set G0 of strokes displayed onthe touchscreen display 41 using the electronic pen 50. The pointingoperation may be, for example, an operation of continuously pressing theposition of “eggs” (group G4) in FIG. 6B or FIG. 7B.

In step S22, the user terminal 40 reads the post-assignment ink storedin the memory 47 and acquires the semantic attribute of group G4 pointedin step S21. Specifically, the user terminal 40 analyzes the groupingdata of the post-assignment ink, thereby identifying a group IDcorresponding to the position pointed at by the electronic pen 50. Then,the user terminal 40 analyzes the semantic data of the post-assignmentink, thereby identifying the semantic attribute corresponding to thegroup ID.

In step S23, the user terminal 40 receives, with the sensor electrode43, a data signal transmitted from the electronic pen 50, therebyacquiring the pen ID of the electronic pen 50.

In step S24, the user terminal 40 transmits the semantic data acquiredin step S22, with the pen ID acquired in step S23 associated therewith,to the content server 30. That is, the user terminal 40 requests thecontent server 30 to perform a search for content.

In step S25, the content server 30 receives the data transmitted fromthe user terminal 40 in step S24, thereby acquiring the pen ID and thesemantic attribute.

In step S26, the content server 30 performs a search for content usingthe semantic attribute and the pen ID acquired in step S25.Specifically, the content server 30 sequentially performs [1] a searchfor content using the pen ID, and [2] a search within the content usingthe semantic attribute. Here, the pen ID serves as identificationinformation (hereinafter, a use ID) for determining whether or not useof content is permitted. As the use ID, not only the pen ID but alsoidentification information of user terminal 40 (hereinafter, a terminalID), identification information of the user (hereinafter, a user ID), ora combination thereof may be used.

FIG. 9A is a diagram illustrating an example of the data structure ofelectronic pen information 62. The electronic pen information 62indicates correspondence relationships between a “pen ID,” which is theidentification information of the electronic pen 50, a “school” to whichthe user belongs, and a “year/class” to which the user belongs. The penID corresponds to the use ID that manages usage of content. Examples ofthe “school” include a primary school, a junior high school, a highschool, a college, a preparatory school, and so on. For the“year/class,” various years or classes can be chosen according to thetype of the school.

FIG. 9B is a diagram illustrating an example of the data structure ofcontent management information 64. The content management information 64indicates correspondence relationships between a “school” to which theuser belongs, a “year/class” to which the user belongs, and “contenttypes” that are available for use. Each of the “school” and the“year/class” is basically defined in the same manner as in theelectronic pen information 62. Further, examples of the content typesinclude titles of books, including textbooks, reference books, andworkbooks, names of study subject, names of publishers, and so on.

The content server 30 reads the electronic pen information 62 and thecontent management information 64, and uses the “school” and the“year/class” as search keys to thereby acquire at least one type ofcontent corresponding to the pen ID. Then, employing any of a variety ofsearch methods, the content server 30 acquires, from within the contentbeing searched, information (hereinafter referred to as “related contentC1”) that matches or approximately matches the “(value of) semanticattribute” that is used as a search key.

In step S27 in FIG. 8, the content server 30 transmits the relatedcontent C1 as a search result of step S26, with the pen ID associatedtherewith, to the user terminal 40.

In step S28, the user terminal 40 receives the data transmitted from thecontent server 30 in step S27, thereby acquiring the related content C1and the pen ID.

In step S29, the user terminal 40 temporarily stores the related contentC1 acquired in step S28 in the memory 47.

In step S30, the user terminal 40 displays the related content C1, whichis retrieved by the search, and the set G0 of strokes simultaneously onthe touchscreen display 41.

FIGS. 10A and 10B are transition diagrams illustrating a third statechange in the display of the user terminal 40. In more detail, FIG. 10Aillustrates the state of the display before the related content C1appears, while FIG. 10B illustrates the state of the display after therelated content C1 appears. As will be understood from these figures,the related content C1 is displayed near (here, at the lower right of)“eggs” as a result of the pointing operation performed on the positionof “eggs.” The related content C1 includes “eggs” written in English and“Imi: tamago” written in Japanese.

The user can easily recognize that a Japanese translation of the Englishword “eggs,” which has been pointed at using the electronic pen 50, is“tamago (plural).” Thus, the digital ink processing system 10 provides alearning support service that facilitates learning with increasedefficiency.

Here, the administrator of the electronic notebook may decide to lendthe user in school the electronic notebook while prohibiting the userfrom taking the electronic notebook out of school. In such cases, byassociating the use ID of the content with the pen ID, it is possible toallow the user to receive the above-described learning support serviceeven at home.

FIG. 11A illustrates a case in which a user terminal 40B is differentfrom a user terminal 40A on which handwriting input is performed usingthe electronic pen 50. For example, the user is able to edit the digitalink Ink using the same electronic pen 50, which has been brought backhome from school. Alternatively, the digital ink processing system 10may be configured to refer to the terminal ID included in the “devicedata” of the digital ink Ink, and if the user terminal 40B does notmatch the terminal ID restrict or prohibit editing of the digital inkInk on the user terminal 40B.

Meanwhile, as illustrated in FIG. 11B, when the user has pointed at arelevant region using the electronic pen 50, the user terminal 40B isallowed to display the related content C1 on the touchscreen display 41as in the case of the user terminal 40A. Thus, even in a situation inwhich it is difficult to freely move the user terminal 40A, the user isable to receive the learning support service at a variety of places bycarrying his or her own electronic pen 50.

Refined Examples of Operation

Hereinafter, refined examples of the operation of the digital inkprocessing system 10 will be described with reference to FIGS. 12A to13B.

First Refined Example

It is conceivable that, as the learning of the user progresses and hisor her degree of achievement increases, information that the userdesires to know will gradually change. Accordingly, the user terminal 40may be configured to display different related content C1 and C2depending on the degree of achievement of the user. For example, in thesequence diagram of FIG. 8, additional step S31 may be inserted betweensteps S23 and S24.

In step S31 in FIG. 8, the user terminal 40 performs a search in thesemantic data with the “value” (for example, eggs) of the semanticattribute acquired in step S22 used as a keyword. The target of thissearch is not limited to the digital ink Ink being edited but mayinclude digital ink Ink stored in the memory 47 of the user terminal 40itself or in the digital ink server 20. For example, the user terminal40 may count the number of “values” that match or approximately matchthe keyword, and calculate the number as the “degree of occurrence.”Thereafter, in steps S24 and S25, transmission and reception of dataincluding the degree of occurrence are performed between the userterminal 40 and the content server 30.

In step S26, the content server 30 performs a search for content usingthe pen ID, the semantic attribute, and the degree of occurrenceacquired in step S25. Specifically, the content server 30 sequentiallyperforms [1] a search for content using the pen ID, [2] a selection ofcontent using the degree of occurrence, and [3] a search in the contentusing the semantic attribute.

FIG. 12A is a diagram illustrating an example of the data structure of adetermination table 66. The determination table 66 indicatescorrespondence relationships between the “number of occurrences,” whichindicates the degree of occurrence, and the “degree of achievement” oftypical users. In the example of this figure, the determination table 66describes determination criterions that determine [1] the degree ofachievement to be low when the number of occurrences is less than 9, [2]the degree of achievement to be intermediate when the number ofoccurrences is 10 or more and less than 20, and [3] the degree ofachievement to be high when the number of occurrences is 20 or more.Note that, although the number of occurrences is used as an example ofthe degree of occurrence in this figure, the frequency of occurrencesmay be used in combination therewith or alternatively.

The content server 30 identifies the degree of achievement of the userfrom the degree of occurrence using the determination table 66, andthereafter selects content that matches the degree of achievement fromamong a plurality of types of content corresponding to the pen ID.Suppose, for example, that the related content C2, which is differentfrom the related content C1 illustrated in FIG. 10B, has been acquiredfor a user who has a higher degree of achievement. Thereafter, steps S27to S29 are performed in a manner similar to that described above.

In step S30, the user terminal 40 displays the related content C2retrieved by the search on the touchscreen display 41 with the set G0 ofstrokes. As illustrated in FIG. 12B, the related content C2 is displayedtogether at a position near “eggs,” on which the pointing operation isperformed. The related content C2 includes a string of characters of anexample sentence “Birds lay eggs.” in which the English word “eggs” isused.

As described above, when different related content C1 or C2 is displayeddepending on the degree of occurrence of the semantic attribute in thesame use ID (here, pen ID), suitable information can be presented toeach individual user having a different degree of achievement. Note thatthere is a high correlation between the degree of occurrence and thedegree of interest of the user, and therefore, it may be useful toperform a display similar to the display in the case of the degree ofachievement described above.

Second Refined Example

Electronic notebooks are respectively used by individuals, and acondition may be undesirable in which an electronic notebook of oneperson can be freely edited by another person. Accordingly, the userterminal 40 may be configured to be capable of changing editing rightsdepending on the pen ID of an electronic pen 50A or 50B.

As illustrated in FIG. 13A, the pen ID (for example, ID=002) of theelectronic pen 50B, which has been recognized by the user terminal 40,is different from the pen ID (for example, ID=001) of the electronic pen50A, which has been used in the handwriting input. In this case, theuser terminal 40 displays, on the touchscreen display 41, a message 70that indicates that editing of the digital ink Ink is restricted. In thesituation in which the message 70 is displayed, the user terminal 40prohibits or restricts editing of the digital ink Ink using theelectronic pen 50B. Here, “prohibiting” means an operation of rejectingan operation related to editing. Meanwhile, “restricting” means anoperation of disabling some of functions related to editing.

Meanwhile, as illustrated in FIG. 13B, when the user points at therelevant region using the electronic pen 50B, the user terminal 40displays the related content C1 on the touchscreen display 41 as in thecase of the electronic pen 50A. Thus, while the user can prevent his orher electronic notebook from being freely edited by others, any otherperson could receive content citation service thereon.

[Charging Flow of Learning Support Service]

Here, along with provision of the above-described learning supportservice, charging and payment are made as for [1] a usage fee of asemantic interpretation service and [2] a usage fee of the contentcitation service. Hereinafter, a flow of charging performed between thedigital ink server 20, the content server 30, the user terminal 40, anda business entity-side server 80 will be described.

Here, the business entity-side server 80 is a server managed by abusiness entity (e.g., a publisher) that provides content. Note that,although it is assumed for the ease of description that the userterminal 40 is involved in the flow of charging, a school-side server(not illustrated) that manages the individual user terminals 40 may beadditionally or alternatively involved in the flow of charging.

First Example

FIG. 14 is a schematic diagram illustrating a first example of the flowof charging made at the time of the learning support service.

-   -   [T11] The user terminal 40 makes access to the digital ink        server 20 each time the semantic interpretation service is used.    -   [T12] The digital ink server 20 totals usage of the semantic        interpretation service for each user on each cutoff date for        usage fees, and performs a charging process of charging the        usage fees of the service in accordance with a predetermined        charging rule.    -   [T13] The digital ink server 20 provides, to each user terminal        40, a notice of demand for the usage fee determined by the        charging process. When the user, receiving this notice, makes a        payment by a due date, the payment of the usage fee by the user        is completed.    -   [T14] Meanwhile, the user terminal 40 makes access to the        content server 30 each time the content citation service is        used.    -   [T15] The content server 30 totals usage of content involved in        the content citation service for each user on each cutoff date        for usage fees.    -   [T16] The content server 30 provides, to the business        entity-side server 80, a notice of usage of content by each        user.    -   [T17] The business entity-side server 80 performs a charging        process of charging the usage fees of the content in accordance        with a predetermined charging rule on the basis of a result of        the totaling of the usage.    -   [T18] The business entity-side server 80 provides, to each user        terminal 40, a notice of demand for the usage fee determined by        the charging process. When the user, receiving this notice,        makes a payment by a due date, the payment of the usage fee by        the user is completed.    -   [T19] The business entity-side server 80 provides, to the        content server 30, a notice of payment of a fee for provision of        a service platform in accordance with a predetermined contract        rule (e.g., a variable fee system that charges a fee in        proportion to the usage fees of the content).

Second Example

FIG. 15 is a schematic diagram illustrating a second example of the flowof charging made at the time of the learning support service.

-   -   [T21] The user terminal 40 makes access to the digital ink        server 20 each time the semantic interpretation service is used.    -   [T22] The digital ink server 20 totals usage of the semantic        interpretation service for each user on each cutoff date for        usage fees, and performs a charging process of charging the        usage fees of the service in accordance with a predetermined        charging rule.    -   [T23] The digital ink server 20 provides, to each user terminal        40, a notice of demand for the usage fee determined by the        charging process. When the user, receiving this notice, makes a        payment by a due date, the payment of the usage fee by the user        is completed.    -   [T24] Meanwhile, the user terminal 40 makes access to the        content server 30 each time the content citation service is        used.    -   [T25] The content server 30 totals usage of content involved in        the content citation service for each user on each cutoff date        for usage fees, and performs a charging process of charging the        usage fees of the content in accordance with a predetermined        charging rule.    -   [T26] The content server 30 provides, to each user terminal 40,        a notice of demand for the usage fee determined by the charging        process. When the user, receiving this notice, makes a payment        by a due date, the payment of the usage fee by the user is        completed.    -   [T27] The content server 30 provides, to the business        entity-side server 80, a notice of payment of an amount obtained        by subtracting a fee for provision of the service platform from        the usage fees of the content.

[Example Modifications] It is needless to say that the presentdisclosure is not limited to the above-described embodiments, and thatvarious modifications can be made thereto without departing from theprinciples of the disclosure. Also note that various features thereofmay be combined optionally as long as no technical conflict arises.

Although, in the above-described embodiments, the digital ink processingsystem 10 is configured to provide a learning support service to supporteducation for students, the type of the service and target users are notlimited to the above-described examples.

Although the digital ink server 20 performs the grouping of strokes andthe assignment of semantic attributes in the above-describedembodiments, the user terminal 40 may be configured to be capable ofperforming the grouping or both the grouping and the assignment in placeof the digital ink server 20. For example, the user terminal 40 itselfmay analyze the digital ink Ink to acquire the semantic attributes, orthe user terminal 40 may acquire the semantic attributes through amanual input operation by the user.

Although the digital ink Ink in which the semantic data is defined hasbeen used as an example in the description of the above-describedembodiments, digital ink Ink in which the semantic data is not definedmay be used. In this case, the digital ink Ink and the semantic data maybe associated with each other and managed, so that an operation similarto that of the above-described embodiments can be realized.

[Summary of Embodiments] As described above, the digital ink processingsystem 10 includes the electronic pen 50, and the user terminal 40 thatis configured to be capable of accepting a writing operation by theelectronic pen 50 through the touchscreen display 41 and generating thedigital ink Ink describing the set G0 of strokes. The user terminal 40enables the pointing operation of the electronic pen 50 relative togroups G2 to G5 of strokes, which satisfy the determination conditionconcerning the grouping of strokes, within the set G0 displayed on thetouchscreen display 41, while deferring the enablement of the pointingoperation relative to group G1 of strokes, which does not satisfy thedetermination condition. If any enabled pointing operation is received,the user terminal 40 performs a search for content related to thesemantic attribute of the specified one of groups G2 to G5, or requestsan external entity to perform the search, and displays the relatedcontent C1 or C2 retrieved by the search on the touchscreen display 41with the set G0.

In addition, according to a digital ink processing method and a digitalink processing program, the user terminal 40 performs the step (S13) ofenabling the pointing operation of the electronic pen 50 relative togroups G2 to G5 of strokes which satisfy the determination conditionconcerning the grouping of strokes within the set G0 of strokesdisplayed on the touchscreen display 41, while deferring the enablementof the pointing operation relative to group G1 of strokes which does notsatisfy the determination condition, and the step (S30) of, if anyenabled pointing operation is received (S21), performing a search forcontent related to the semantic attribute assigned to groups G2-G5 orrequesting an external entity to perform the search (S24), anddisplaying the related content C1 or C2 retrieved by the search on thetouchscreen display 41 along with the set G0.

As described above, because the pointing operation of the electronic pen50 is enabled only with respect to groups G2 to G5 of strokes whichsatisfy the determination condition concerning the grouping of strokes,a search for content will be permitted only with respect to groups G2 toG5 which are highly likely to have meanings as collections when thedetermination condition is properly set. This makes it possible topresent, to the user, useful and highly relevant information as a searchresult when a search is performed using the digital ink Ink.

In addition, the user terminal 40 may cause groups G2 to G5, for whichthe pointing operation has been enabled, to be highlighted as comparedto before the enablement. This makes it easier for the user to visuallyrecognize the enablement of the pointing operation and the positionsthereof.

Further, the determination condition may be assignment of the semanticattribute to groups G2-G5 of strokes. This makes it possible to performa search using the assigned semantic attribute as a search key so thatthe user can obtain a more useful search result.

Furthermore, while the digital ink server 20 analyzes the digital inkInk transmitted from the user terminal 40 and thus assigns the semanticattributes to groups G2 to G5, the determination condition may beacquisition of the semantic data representative of the semanticattributes from the digital ink server 20. The assignment of thesemantic attributes by the digital ink server 20 in place of the userterminal 40 reduces the analytical load on the user terminal 40.

Furthermore, the determination condition may be acceptance of a useroperation of adding a mark or an annotation to groups G2-G5. This makesit possible for a user who performs the operation to specify, throughthe mark or annotation, a collection of strokes, in which the user findsmeaning, leading to an increased likelihood that a more accuratesemantic attribute will be obtained.

Furthermore, the content server 30 may store content in association withuse IDs, and the user terminal 40 may transmit data including the use IDand the semantic attribute to the content server 30 and may request asearch to thereby acquire, from the content server 30, the relatedcontent C1 or C2 which is related to the semantic attribute and which isusable.

Furthermore, the user terminal 40 may acquire different related contentC1 or C2 depending on the degree of occurrence of the semantic attributebased on the same use ID. When the degree of occurrence, which is highlycorrelated with the degree of achievement or degree of interest of theuser, is taken into account, suitable information can be selected andpresented to each of users who may vary in the degree of achievement ordegree of interest.

Furthermore, the use ID may be the pen ID for identifying the electronicpen 50. This makes it possible to associate the electronic pen 50 withcontent available for use, making it possible to present the samerelated content C1 or C2 when the same electronic pen 50 is used,regardless of which of the user terminals 40A or 40B may be used.

Furthermore, in the case where the digital ink Ink has been generatedusing the electronic pen 50A that is assigned a first pen ID, and theelectronic pen 50B is used that is assigned a second pen ID differentfrom the first pen ID, the user terminal 40 may prohibit or restrictediting of the digital ink Ink while accepting the pointing operation bythe electronic pen 50B. Thus, while the digital ink Ink generated by theuser is prohibited from being freely edited, the same related content C1or C2 as will be presented to the user will be presented even to aperson other than the user.

Furthermore, the content server 30 may perform a charging process inaccordance with usage of content for the user of the content and abusiness entity that provides the content. Thus, management of thecontent and management of usage fees can be accomplished in acentralized manner by a single apparatus.

DESCRIPTION OF REFERENCE SYMBOLS

-   -   10: Digital ink processing system    -   20: Digital ink server    -   30: Content server    -   40, 40A, 40B: User terminal    -   41: Touchscreen display    -   50, 50A, 50B: Electronic pen    -   80: Business entity-side server    -   C1, C2: Related content    -   G0: Set    -   G1 to G5: Group    -   Ink: Digital ink

1. A digital ink processing system comprising: an electronic pen; and atablet that generates digital ink describing a stroke or strokesaccording to an input made by the electronic pen through a display,wherein the tablet includes a processor that enables a pointingoperation of the electronic pen relative to the stroke or strokes, afterenabling the pointing operation of the electronic pen relative to thestroke or strokes, performs a search for content related to a semanticattribute of the stroke or strokes pointed at, or requests an externalserver to perform the search, and performs control so as to displaycontent retrieved by the search on the display with the stroke orstrokes.
 2. The digital ink processing system according to claim 1,wherein the processor performs control so as to cause the stroke orstrokes for which the pointing operation has been enabled to behighlighted on the display as compared to before the enablement.
 3. Thedigital ink processing system according to claim 1, wherein theprocessor enables the pointing operation of the electronic pen relativeto the stroke or strokes to which the semantic attribute has beenassigned.
 4. The digital ink processing system according to claim 3,further comprising: a digital ink server that analyzes the digital inktransmitted from the tablet and assigns the semantic attribute to thestroke or strokes, wherein the processor enables the pointing operationof the electronic pen relative to the stroke or strokes for whichsemantic data representative of the semantic attribute has been acquiredfrom the digital ink server.
 5. The digital ink processing systemaccording to claim 1, wherein the processor enables the pointingoperation of the electronic pen relative to the stroke or strokes onwhich a user operation of adding a mark or an annotation has beenperformed.
 6. The digital ink processing system according to claim 1,further comprising: a content server that stores content in associationwith use IDs (identifications), wherein the processor transmits data,which includes the use ID and the semantic attribute, to the contentserver and request a search to acquire, from the content server, contentthat relates to the semantic attribute and that is usable.
 7. Thedigital ink processing system according to claim 6, wherein theprocessor acquires a different one of the content depending on a degreeof occurrence of the semantic attribute based on the same use ID.
 8. Thedigital ink processing system according to claim 6, wherein the use IDis a pen ID that identifies the electronic pen.
 9. The digital inkprocessing system according to claim 8, wherein in a case where thedigital ink is generated using a first electronic pen that is assigned afirst pen ID, and a second electronic pen is used that is assigned asecond pen ID different from the first pen ID, the processor prohibitsor restrict editing of the digital ink while accepting the pointingoperation.
 10. The digital ink processing system according to claim 6,wherein the content server performs a charging process in accordancewith usage of content, for a user of the content and a business entitythat provides the content.
 11. A digital ink processing method executedby a processor configured to generate digital ink describing a stroke orstrokes according to an input made by an electronic pen through adisplay, the method comprising: enabling a pointing operation of theelectronic pen relative to the stroke or strokes; after enabling thepointing operation of the electronic pen, performing a search forcontent related to a semantic attribute of the stroke or strokes pointedat, or requesting an external server to perform the search; andperforming control so as to display content retrieved by the search onthe display with the stroke or strokes.
 12. A digital ink processingprogram for causing a processor, which generates digital ink describinga stroke or strokes according to an input made by an electronic penthrough a display, to execute: a step of enabling a pointing operationof the electronic pen relative to the stroke or strokes; a step of,after enabling the pointing operation of the electronic pen, performinga search for content related to a semantic attribute of the stroke orstrokes pointed at, or requesting an external server to perform thesearch; and a step of performing control so as to display contentretrieved by the search on the display with the stroke or strokes.